{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2019 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/workshops/Transfer_Learning/Idiomatic%20Programmer%20-%20handbook%203%20-%20Codelab%203.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idiomatic Programmer Code Labs\n",
    "\n",
    "## Code Labs #3 - Get Familiar with Transfer Learning\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Handbook 3/Part 12: Prebuilt Models & Transfer Learning\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Use a prebuilt model and fully retrain.\n",
    "    2. Use a prebuilt model and do transfer learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retraining with a Prebuilt Model\n",
    "\n",
    "We will use a stock (prebuilt) ResNet50 model from the Keras model zoo repository, and:\n",
    "\n",
    "    1. Replace the input layer to match the CIFAR-10 dataset.\n",
    "    2. Replace the output layer to match the CIFAR-10 dataset.\n",
    "    3. Leave weights to the default initialization (untrained).\n",
    "    4. Train the model on CIFAR-10\n",
    "\n",
    "You fill in the blanks (replace the ??), make sure it passes the Python interpreter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import Model\n",
    "from keras.layers import Dense\n",
    "\n",
    "# We will use a stock ResNet50 model\n",
    "from keras.applications import ResNet50\n",
    "\n",
    "# Let's get a version of the ResNet50, which is missing the classifier (we will replace it), \n",
    "# has default initialization and input size will be for the CIFAR-10 dataset\n",
    "# HINT: the top is the last layer (classifier) in the model, we don't want to include it.\n",
    "model = ResNet50(include_top=??, input_shape=(32, 32, 3), pooling='avg')\n",
    "\n",
    "# Add a classifier for 10 classes\n",
    "# HINT: the input to this layer is the output from the above model.\n",
    "outputs = Dense(10, activation='softmax')(??)\n",
    "model = Model(model.input, outputs)\n",
    "\n",
    "# Compile the model for training\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import to_categorical\n",
    "from keras.datasets import cifar10\n",
    "import numpy as np\n",
    "\n",
    "# Let's use the CIFAR-10 dataset\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# Normalize the pixel data\n",
    "x_train = (x_train / 255.0).astype(np.float32)\n",
    "x_test  = (x_test  / 255.0).astype(np.float32)\n",
    "\n",
    "# One-hot encode the labels\n",
    "y_train = to_categorical(y_train)\n",
    "y_test  = to_categorical(y_test)\n",
    "\n",
    "# ResNet50 will take along time to wrong an epoch (vs. our simple ConvNet)\n",
    "# Will will use just a small amount for brevity (2%)\n",
    "x_tmp = x_train[0:1000]\n",
    "y_tmp = y_train[0:1000]\n",
    "\n",
    "# Let's now train the prebuilt ResNet50 with the CIFAR-10 data\n",
    "# HINT: You've been using this method a lot.\n",
    "model.??(x_tmp, y_tmp, epochs=5, batch_size=32, validation_split=0.1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transfer Learning with a Prebuilt Model\n",
    "\n",
    "Okay, our accuracy for the above is poor. Well, we only used 2% of the training data, and ran just a few epochs.\n",
    "\n",
    "This time, let's go ahead and try transfer learning:\n",
    "\n",
    "    1. Repeat getting and modifying the prebuilt ResNet50 as before.\n",
    "    2. This time though initialize it with the learned weights from imagenet.\n",
    "    3. Freeze the already trained ResNet50 layers.\n",
    "    4. Train the top (classifier) layer we added.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Let's get a version of the ResNet50, which is missing the classifier (we will replace it), \n",
    "# has default initialization and input size will be for the CIFAR-10 dataset\n",
    "# HINT: the parameter name refers to the weights\n",
    "model = ResNet50(include_top=False, input_shape=(32, 32, 3), pooling='avg', ??='imagenet')\n",
    "\n",
    "# Freeze the weights of the remaining layer\n",
    "# HINT: the name of property (field) is something to do with being trainable.\n",
    "for layer in model.layers:\n",
    "    layer.?? = False\n",
    "    \n",
    "\n",
    "# Add a classifier for 10 classes\n",
    "outputs = Dense(10, activation='softmax')(model.output)\n",
    "model = Model(model.input, outputs)\n",
    "\n",
    "# Compile the model for training\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's now train the prebuilt ResNet50 with the CIFAR-10 data\n",
    "# This will train a lot faster since we are only updated the classifier layer -- so let's use all the data.\n",
    "model.fit(x_train, y_train, epochs=5, batch_size=32, validation_split=0.1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Code Lab"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
